Joint entropy

 ( Entropy And Information ) 

In information theory, joint entropy is a measure of the uncertainty associated with a set of random variables.

---

Definition The joint Shannon entropy (in ) of two discrete random variable $X$ and $Y$ with images $\backslash mathcal\; X$ and $\backslash mathcal\; Y$ is defined as

(2006). 9780471241959, Wiley. ISBN 9780471241959

$\backslash Eta(X,Y)\; =\; -\backslash sum\_\{x\backslash in\backslash mathcal\; X\}\; \backslash sum\_\{y\backslash in\backslash mathcal\; Y\}\; P(x,y)\; \backslash log\_2P(x,y)$

where $x$ and $y$ are particular values of $X$ and $Y$, respectively, $P(x,y)$ is the joint probability of these values occurring together, and $P(x,y)\; \backslash log\_2P(x,y)$ is defined to be 0 if $P(x,y)=0$.

For more than two random variables $X\_1,\; ...,\; X\_n$ this expands to

$\backslash Eta(X\_1,\; ...,\; X\_n)\; =$

-\sum_{x_1 \in\mathcal X_1} ... \sum_{x_n \in\mathcal X_n} P(x_1, ..., x_n) \log_2P(x_1,

where $x\_1,...,x\_n$ are particular values of $X\_1,...,X\_n$, respectively, $P(x\_1,\; ...,\; x\_n)$ is the probability of these values occurring together, and $P(x\_1,\; ...,\; x\_n)\; \backslash log\_2P(x\_1,$ is defined to be 0 if $P(x\_1,\; ...,\; x\_n)=0$.

---

Properties

---

Nonnegativity The joint entropy of a set of random variables is a nonnegative number.

$\backslash Eta(X,Y)\; \backslash geq\; 0$

$\backslash Eta(X\_1,\backslash ldots,\; X\_n)\; \backslash geq\; 0$

---

Greater than individual entropies The joint entropy of a set of variables is greater than or equal to the maximum of all of the individual entropies of the variables in the set.

$\backslash Eta(X,Y)\; \backslash geq\; \backslash max\; \backslash left\backslash Eta(X),\backslash Eta(Y)$

$\backslash Eta\; \backslash bigl(X\_1,\backslash ldots,\; X\_n\; \backslash bigr)\; \backslash geq\; \backslash max\_\{1\; \backslash le\; i\; \backslash le\; n\}$

   \Bigl\{ \Eta\bigl(X_i\bigr) \Bigr\}
     

---

Less than or equal to the sum of individual entropies The joint entropy of a set of variables is less than or equal to the sum of the individual entropies of the variables in the set. This is an example of subadditivity. This inequality is an equality if and only if $X$ and $Y$ are statistically independent.

$\backslash Eta(X,Y)\; \backslash leq\; \backslash Eta(X)\; +\; \backslash Eta(Y)$

$\backslash Eta(X\_1,\backslash ldots,\; X\_n)\; \backslash leq\; \backslash Eta(X\_1)\; +\; \backslash ldots\; +\; \backslash Eta(X\_n)$

---

Relations to other entropy measures

Joint entropy is used in the definition of conditional entropy

$\backslash Eta(X|Y)\; =\; \backslash Eta(X,Y)\; -\; \backslash Eta(Y)\backslash ,$,

and

$\backslash Eta(X\_1,\backslash dots,X\_n)\; =\; \backslash sum\_\{k=1\}^n\; \backslash Eta(X\_k|X\_\{k-1\},\backslash dots,\; X\_1)$.

For two variables $X$ and $Y$, this means that

$\backslash Eta(X,Y)\; =\; \backslash Eta(Y)\; +\; \backslash Eta(X|Y)\; =\; \backslash Eta(X)\; +\; \backslash Eta(Y|X)$.

Joint entropy is also used in the definition of mutual information

$\backslash operatorname\{I\}(X;Y)\; =\; \backslash Eta(X)\; +\; \backslash Eta(Y)\; -\; \backslash Eta(X,Y)\backslash ,$.

In quantum information theory, the joint entropy is generalized into the joint quantum entropy.

---

Joint differential entropy

---

Definition

The above definition is for discrete random variables and just as valid in the case of continuous random variables. The continuous version of discrete joint entropy is called joint differential (or continuous) entropy. Let $X$ and $Y$ be a continuous random variables with a joint probability density function $f(x,y)$. The differential joint entropy $h(X,Y)$ is defined as

$h(X,Y)\; =\; -\backslash int\_\{\backslash mathcal\; X\; ,\; \backslash mathcal\; Y\}\; f(x,y)\backslash log\; f(x,y)\backslash ,dx\; dy$

For more than two continuous random variables $X\_1,\; ...,\; X\_n$ the definition is generalized to:

$h(X\_1,\; \backslash ldots,X\_n)\; =\; -\backslash int\; f(x\_1,\; \backslash ldots,x\_n)\backslash log\; f(x\_1,\; \backslash ldots,x\_n)\backslash ,dx\_1\; \backslash ldots\; dx\_n$

The integral is taken over the support of $f$. It is possible that the integral does not exist in which case we say that the differential entropy is not defined.

---

Properties

As in the discrete case the joint differential entropy of a set of random variables is smaller or equal than the sum of the entropies of the individual random variables:

$h(X\_1,X\_2,\; \backslash ldots,X\_n)\; \backslash le\; \backslash sum\_\{i=1\}^n\; h(X\_i)$

The following chain rule holds for two random variables:

$h(X,Y)\; =\; h(X|Y)\; +\; h(Y)$

In the case of more than two random variables this generalizes to:

$h(X\_1,X\_2,\; \backslash ldots,X\_n)\; =\; \backslash sum\_\{i=1\}^n\; h(X\_i|X\_1,X\_2,\; \backslash ldots,X\_\{i-1\})$

Joint differential entropy is also used in the definition of the mutual information between continuous random variables:

$\backslash operatorname\{I\}(X,Y)=h(X)+h(Y)-h(X,Y)$

Categories: Entropy And Information

Post Comment